Image quality for diagnostic resolution digital slide images

ABSTRACT

An improved diagnostic resolution of digital slide images is obtained by scanning a first digital slide image at diagnostic resolution that is then deconvolved into separate images with one stain per image. The single stain images are then enhanced with image adjustments and/or processed with image analysis algorithms. The resulting single image data sets from the image analysis algorithms can then be stored. Additionally, the resulting enhanced single images can be recombined into a second digital slide image at diagnostic resolution that is also enhanced.

RELATED APPLICATION

The present application claims priority to U.S. provisional patentapplication Ser. No. 60/974,337 filed Sep. 21, 2007, which isincorporated herein by reference in its entirety.

BACKGROUND

1. Field of the Invention

The present invention generally relates to the field of digitalpathology and more specifically relates to systems and methods forimproving the image quality of a digital slide image for diagnosticresolution.

2. Related Art

While scientific measures such as modulation transfer function (“MTF”),resolution and the signal to noise ratio (“SNR”) can be used toobjectively quantify image capture devices, it is relatively difficultto objectively assess the quality or interpretability of the imagerydata that is generated by such devices, particularly when the imagerydata is destined to be subjectively examined by a human, for example inthe evaluation of digital slide images in pathology.

Furthermore, the challenges associated with assessing image qualityand/or interpretability are exacerbated for digital slide images thatare read by pathologists because digital slides are extraordinarilylarge, typically multiple gigabytes in size. This significant amount ofimagery data makes it very difficult to examine every area of a digitalslide at full resolution to determine its quality and/orinterpretability. Additionally, the patterns (i.e., the features andclues) that pathologists look for when making a diagnosis vary betweentissue types and this is further confounded because there is no formalagreement among pathologists about the criteria and tasks used to makediagnoses of specific tissue types. Further, analysis of digital slideimages and diagnosis based on digital slide images is made difficult byweak stains that result in faint coloring, strong stains that result inheavy coloring, non-standardized digital image characteristics ofstains, similarly colored stains in a single slide, stains with lowcontrast to each other on a single slide, and poorly physical slidepreparation. Therefore, what is needed is a system and method thatprovides improved quality and/or interpretability of a digital slideimage to overcome these significant problems as described above.

SUMMARY

Described herein are systems and methods for improving the diagnosticresolution of digital slide images. In one embodiment a digital slideimage is received, for example scanned, at diagnostic resolution andthen deconvolved into separate images with one stain per image.Deconvolution can refer to the process of reversing the interaction oftwo or more colored stains on tissue.

The single stain images are then enhanced with image adjustments and/orprocessed with image analysis algorithms. The resulting single imagedata sets from the image analysis algorithms can then be stored.Additionally, the resulting enhanced single images can be recombinedinto a second digital slide image at diagnostic resolution that is alsoenhanced.

In one embodiment of a method for improving diagnosis of digital slideimages, a first image of a tissue sample is scanned at diagnosticresolution. The image of the tissue sample can comprise a plurality ofstains, for example Haematoxylin, Eosin or Diaminobenzidine (“DAB”).Staining procedures used to highlight structures in a tissue or cellusually involve techniques that use dyes such as Haematoxylin, Eosin orDAB. The image of the tissue is deconvloved into a plurality of singleimages and image adjustments are made on one or more of the plurality ofsingle stain images. The one or more of the plurality of single stainimages can then be combined into a second image at diagnosticresolution, where the second image is an improved digital image. In oneembodiment, the plurality of single stains images can remain separatefor viewing.

In some embodiments the image adjustments include digital recolorizationof a stain on a single stain image. This can result in higher qualitydigital images for more objective diagnosis and improve quality of imageanalysis algorithm results. In other embodiments, the one or more of theplurality of single stain images can be processed with one or more imageanalysis algorithms or image quality algorithm. The image qualityalgorithm can successfully separate the individual stains, digitallyadjust each stain independently and recombine them in pairs to presentthe digital slides. The use of the IQ algorithm can produce digitallyenhanced stains, a reduction in samples used, automatic co-location ofmultiple stains, for example hematoxylin and eosin (“H&E”) with H&DAB,thereby accelerating review, saving preparation time and time spentrequeuing the number of glass slides and storage. The image qualityalgorithm can also configure for any two or three stained sample, forexample H&E with H&DAB or H&E&DAB. The image quality algorithm alsoperforms digital stain adjustments for stain normalization includingdilution/concentration, removal of background staining, cellular detailemphasis, chromogen tuning and remapping.

In another embodiment of the system, a server comprises a deconvolutionmodule, a recombination module, an adjustment module and a receivingmodule, for example a scanner module. The server can be configured witha data storage area and a scanner module to scan a first image into thesystem. While the scanner module is included in the server, it can alsobe a separate entity. The deconvolution module is employed to deconvolvea first image into a plurality of single images. The goal ofdeconvolution is to recreate the signals or images as if the slide wereonly stained each stain singularly. Deconvolution occurs after imageacquisition, and uses algorithms to extract information out of, forexample, blurred regions of an image to clean up these regions. Therecombination module can be configured to recombine one or more of theplurality of single stain images into a second image at diagnosticresolution. The adjustment module can be configured to make imageadjustments on one or more of the plurality of single stain images. Inone embodiment, the adjustments can be predefined for a given test andthen automatically applied. In another embodiment the adjustment modulecan be configured to receive adjustment instructions from a user, wherethe user can make adjustments for just the current instance of viewing,for example. In some embodiments the user can make adjustments and savethe adjustments as a default preset for a particular test. In otherembodiments, the adjustments can vary with scan or view resolution.These adjustment instructions can be the basis for image adjustments onone or more of the plurality of single stain images. The adjustments canalso vary with scan or view resolution. The server can also include aprocessor to process one or more of the plurality of single stain imageswith one or more image analysis algorithms or image quality algorithm.The data resulting from the image analysis algorithms can be stored inthe data storage area.

A significant advantage of the described systems and methods is thatthey provide a pathologist or clinician with improved image qualitydigital slides that facilitate certain types of diagnostic orinterpretative tasks. For example, the system and method provides forsingle stain digital adjustments such as boosting weak stains, dilutingstrong stains, removing background staining, standardization of digitalimage characteristics of stains, recolorization of stains (e.g., toprimary colors having higher contrast than actual stains used),personalized staining (e.g., based on preference of an individualpathologist), spatial filtering on single stain images, segmentation andquantification on single stain images (e.g., counting nuclei in aProgesterone_receptor (“PR”), support new multiplex reagent withmultichannel detection or spectral imaging detector (e.g., ER/PR on sameslide and other dual or triple stains—reduces the need for two or morephysical slides to be stained, provides advantage over separate slideswith inexact physical samples) and novel methods of visualization.

In one embodiment, the method for improving diagnosis of digital slideimages includes scanning a first image of a tissue sample at diagnosticresolution where the tissue sample can include a plurality of stains.Deconvolving the first image into a plurality of single images andmaking image adjustments on one or more of the plurality of single stainimages. One or more of the single images can also be combined into asecond image at diagnostic resolution. In some embodiments the resultingimproved digital image can be rated. The image adjustment can be adigital recolorization of a stain on a single stain image whererecolorization can provide (a) higher contrast; (b) personalization; (c)normalization or standardization of stains; and (d) identifyingevents/areas of interest (e.g., rare events, ER/PR co-location). Inanother embodiment the method for improving diagnosis of the digitalslide images includes scanning a first image of a tissue sample atdiagnostic resolution where the tissue sample comprising a plurality ofstains. The first image can be deconvolved into a plurality of singleimages. In addition the one or more of the plurality of single stainimages can be processed with one or more image analysis algorithms. Thedata resulting from the image analysis algorithms can be stored in astorage medium.

Development of New Reagents (Stains): In one embodiment, the reagentscan be optimized for digital scanning at a diagnostic resolution. Anexample of an optimized reagent can be achieved by developing newprotocols for use of conventional and new reagents to optimize thesample for being digitally scanned at a diagnostic resolution (e.g.,multiplex reagents or single reagent). In an embodiment new reagentsystems (for example, stains) can be optimized for digital pathology,e.g., produce better quality digital images when scanned at diagnosticresolution. Among other IQ elements, the process can includedeconvolving the new reagent and also New/optimized reagent protocols incombination with deconvolution. In one embodiment the process caninclude linking the properties of the reagent and the light passingthrough the stained tissue into the pixel sensors on the camera (e.g.,optimizing the reagent(s) into the RGB channels.

Other features and advantages of the present invention will become morereadily apparent to those of ordinary skill in the art after reviewingthe following detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of the present invention, both as to its structure andoperation, may be gleaned in part by study of the accompanying drawings,in which like reference numerals refer to like parts, and in which:

FIG. 1 is a network diagram illustrating an example system for imageinterpretability assessment according to an embodiment of the presentinvention;

FIG. 2 is a block diagram illustrating an example server to performimproved diagnostic resolution for digital slide images according to oneembodiment of the present invention.

FIG. 3 is a block diagram illustrating an example server for imageinterpretability assessment according to an embodiment of the presentinvention.

FIG. 4 is a flow diagram illustrating an example process for improveddiagnostic resolution for digital slide images according to oneembodiment of the present invention.

FIG. 5 is a flow diagram illustrating another example process forimproved diagnostic resolution for digital slide images according to oneembodiment of the present invention.

FIG. 6 is an example of a flow process for enhanced progressiverendering when viewing digital slide images of a tissue according to anembodiment.

FIGS. 7-8 are block diagrams illustrating examples of images processedby an image quality algorithm.

FIGS. 9-12 are diagrams illustrating example regions examined andcollocation of markers and side by side viewing of the reconstructions.

FIG. 13 is a diagram illustrating an example of IQ AlgorithmPresentation Modes.

FIGS. 14-17 are diagrams illustrating examples of multiple image qualitypresentations.

FIG. 18 is a block diagram illustrating an example computer system thatmay be used in connection with various embodiments described herein.

DETAILED DESCRIPTION

Certain embodiments as disclosed herein provide for systems and methodsfor improving diagnostic resolution of digital slide images. Afterreading this description it will become apparent to one skilled in theart how to implement the invention in various alternative embodimentsand alternative applications. However, although various embodiments ofthe present invention will be described herein, it is understood thatthese embodiments are presented by way of example only, and notlimitation. As such, this detailed description of various alternativeembodiments should not be construed to limit the scope or breadth of thepresent invention as set forth in the appended claims.

FIG. 1 is a network diagram illustrating an example system 10 for imageinterpretability assessment according to an embodiment of the presentinvention. In the illustrated embodiment, the system 10 comprises animage server 20 that is communicatively coupled with reviewer device 40and reviewer device 50 via a network 30. The image server 20 isconfigured with a data storage area 25.

The image server 20 can be implemented using a conventional computerdevice and is configured to maintain in the data storage area 25 aplurality of digital slide images, for example digital images ofphysical tissue samples on microscope slides. These digital slides canbe in any of a variety of image formats and data formats. The serveralso maintains in data storage area 25 various program modules that canbe used to facilitate and implement the functions for assessing imageinterpretability. For example, modules for obtaining portions of digitalslide imagery data, modules for executing image analysis algorithms onportions of digital slide imagery data, and other modules can bemaintained by the image server 20 in data storage area 25.

The data storage area 25 can be any sort of internal or external memorydevice and may include both persistent and volatile memories. Thefunction of the data storage area 25 is to maintain data (e.g., imagedata and operational modules) for long term storage and also to provideefficient and fast access to instructions for applications that areexecuted by the server 20.

The server 20 and the reviewers 40 and 50 are all communicativelycoupled with the network 30. The network 30 is configured for datacommunications (e.g., between server 20 and reviewer 40) over a widegeographical area and can be communicatively coupled with one or morepublic or private networks (not shown), which may include thatparticular aggregation of networks commonly known as the Internet.

The reviewers 40 and 50 can also be implemented using a conventionalcomputer device or other communication device with the ability toconnect to the network 30. For example, the reviewers 40 and 50 caninclude any of a variety of communication devices including a wirelesscommunication device, personal digital assistant (“PDA”), personalcomputer (“PC”), laptop computer, PC card, special purpose equipment, orany combination of these and other devices capable of establishing acommunication link over network 30 with the server 20. The reviewers 40and 50 may be configured with data storage areas (not shown) and theirprimary function is to present digital slide information to an analyst(e.g., a pathologist or technician) and pass input from the analyst tothe server 20.

FIG. 2 is a block diagram illustrating an example server 20 forperforming diagnostic resolution for digital slide images. In theillustrated embodiment, the server 20 comprises a deconvolution module11, a recombination module 21 and an adjustment module 31 and a scannermodule 41. The server 20 can be configured with a data storage area 25as described above. In one embodiment, the server can also include animage analysis module.

The scanner module 41 is configured to scan a first image into thesystem. While the scanner module 41 is included in the server 20, it canalso as a separate entity. The deconvolution module 11 is employed todeconvolve a first image into a plurality of single images. The goal ofdeconvolution is to recreate the signal or image as it existed beforethe convolution took place. Deconvolution occurs after imageacquisition, and uses algorithms to extract individual stain images. Therecombination module 21 can be configured to recombine one or more ofthe plurality of single stain images into a second image at diagnosticresolution. The adjustment module 31 configured to make imageadjustments on one or more of the plurality of single stain images. Inone embodiment the adjustment module 31 includes a reviewer 40,illustrated in FIG. 1, which can be configured to receive adjustmentinstructions from a user. These adjustment instructions can be the basisfor image adjustments on one or more of the plurality of single stainimages. The adjustment module can be configured to make imageadjustments on one or more of the plurality of single stain images. Inone embodiment, the adjustments can be predefined for a given test andthen automatically applied. In another embodiment the adjustment modulecan be configured to receive adjustment instructions from a user, wherethe user can make adjustments for just the current instance of viewing,for example. In some embodiments the user can make adjustments and savethe adjustments as a default preset for a particular test. In otherembodiments, the adjustments can vary with scan or view resolution.These adjustment instructions can be the basis for image adjustments onone or more of the plurality of single stain images. In one embodimentthe server further comprises a processor 41 to process one or more ofthe plurality of single stain images with one or more image analysisalgorithms. The data resulting from the image analysis algorithms can bestored in the data storage area 25.

FIG. 3 is a block diagram illustrating an example server 20 for imageinterpretability assessment according to an embodiment of the presentinvention. In the illustrated embodiment, the server 20 comprises a taskmodule 100, an image module 110, a task performance module 120, and animage scoring module 130. The server 20 is again configured with a datastorage area 25 as described above. A set of tasks created in the taskmodule is can be used or implemented in the adjustment module 31 to makeimage adjustments on one or more of the plurality of single stainimages.

The task module 100 is employed to create and maintain one or more setsof tasks. The sets of tasks can be tissue type specific, for example afirst set of tasks may be required for analysis of a tissue sample,while a second set of tasks may be required for analysis of a bloodsample. In certain cases, individual tasks within the various sets oftasks may overlap. The sets of tasks may be established by a standardsbody or other similar group and an individual task, for example, mayinclude the diagnostic tasks and criteria that would be performed usingconventional microscopy by a board-certified pathologist who is familiarwith the particular tissue type and stain that appears in the digitalslide image. The set of tasks created in the task module can be used orimplemented in the adjustment module 31 to make image adjustments on oneor more of the plurality of single stain images.

Because a digital slide provides instant access to many intermediateimage resolution levels—from thumbnail image (lowest resolution image)to full-resolution baseline image (highest resolution quality)—a moremeaningful rating scale is created by selecting tasks that can beaccomplished at high image resolution, e.g., using a microscope at highpower (for example, with a 40×/0.75 objective lens). Advantageously, theimage quality rating is however improved by utilizing the IQ Algorithm.The selection of criteria and tasks is tissue-specific and includingtasks that can be accomplished at lower image resolutions may beappropriate for some tissue types (e.g., dermatopathology). Bycomparison, an image interpretability rating scale aimed athematopatholgy specimens, should reflect criteria and tasks thatcorrespond to oil-immersion resolutions; for example using a 100×/1.4objective lens. In general, higher resolution images have higher imagequality when focus and illumination and other image capture attributesare relatively equal.

A list of potential tasks and criteria is provided in the table belowfor a prostate cancer specimen:

-   1. Identify areas of overlapping cores-   2. Identify red blood cells in vessels-   3. Identify corpora amylacea-   4. Distinguish between a Gleason grade 3 and Gleason grade 5 pattern-   5. Distinguish between nerve tissue and stroma-   6. Identify individual muscle bundles-   7. Detect the “pink-colored” secretions of normal prostate glands-   8. Detect the presence of the “amphophilic cytoplasm” of malignant    prostate glands-   9. Identify the basal layer of a prostate gland-   10. Detect the presence of nucleoli-   11. Count the number of nuclei within a gland-   12. Detect the borders of luminal cells-   13. Detect the presence of “blue-tinged mucus-   14. Detect the presence of mitotic figures-   15. Count the number of endothelial nuclei within a vessel-   16. Detect the presence of lipofuschin/pigment-   17. Approximate the diameter of a nucleus to within 2 μm-   18. Detect clear cut nuclear envelope borders-   19. Identify individual nerve fibers-   20. Detect the concavity of individual red blood cells-   21. Identify the individual lobes of a neutrophils-   22. Identify individual striations in muscle fibers-   23. Distinguish between the nucleus of stromal fibroblast and a    lymphocyte-   24. Distinguish between “normal” and “abnormal” mitotic figures-   25. Identify individual nucleoli within a multi-nucleolated cell

In one embodiment, the criteria and tasks for specific tissue types areestablished by consensus among pathologists, with some criteria beingcommon among different tissue types. These criteria are then provided tothe task module 100 and stored in the data storage area 25. While somepathologists may disagree about including or excluding certain tasks orcriteria, organizations like the College of American Pathologists(“CAP”) may advantageously publish and standardize the generallyaccepted tasks and criteria for different tissue types.

The image module 110 is configured to obtain imagery data from a digitalslide. The digital slide may be stored locally or remotely and may beaccessed through a separate database or be integrated with a databaseunder the operation of the image module 110. Starting from scratch, theimage module may facilitate the scanning of a glass slide to create adigital slide image at the required resolution for accomplishing thecriteria and tasks for the particular tissue type. Alternatively, theimage module 110 may obtain imagery data at the required resolution viaa network server, remote storage area, or local data storage area 25. Inone embodiment using prostate cancer biopsies, the required scanningresolution is achieved by use of a 40×/0.75 objective lens, whichcorresponds to a 0.25 μm per pixel scanning resolution.

The task performance module 120 is configured to manage the performanceof the various tasks that are required in the set of tasks for theparticular tissue type. In one embodiment, the task performance moduleprovides portions of imagery data from the digital slide image to thedisplay screen of a reviewer device so that an analyst (e.g., atechnician or pathologist) may analyze the portions of imagery data toperform each task in the set of tasks corresponding to the particulartissue type. The task performance module 120 also records the success orfailure of the analyst in performing the task. This may be accomplishedby presenting a dialogue box to the analyst with a binary inputcapability that allows the analyst to register whether the task wassuccessfully performed or not successfully performed.

Alternatively, or in combination, the task performance module may employone or more computer implemented image processing algorithms that arecapable of analyzing imagery data from a digital slide to perform aparticular task. For example, an image processing algorithm capable ofcounting the number of cells in a specimen may be used to determinewhether there is sufficient contrast and resolution in the imagery datato separately identify individual cells. Thus, successfully being ableto count the number of cells in a tissue sample may result in the taskperformance module 120 recording as successful the task for being ableto separately identify individual cells. Advantageously, as moresophisticated and robust image processing algorithms are developed,fewer tasks may need to be performed by analysts, thus decreasing thetime to score digital slide images and improving the reliability ofpredictable and repeatable scoring through elimination of the humanelement.

In one embodiment, the task performance module 120 measures an analyst'sability to perform each of the diagnostic tasks and criteria in the tasklist on representative portions of imagery data from a digital slide.This is accomplished by the task performance module 120 by selectingrandom portions of imagery data from the digital slide (so as to berepresentative of the entire area of the digital slide) or alternativelyby selecting portions of imagery data from meaningful regions of theslide, which can be identified using a microscope (e.g., manually) orimage processing algorithm (e.g., computer aided/automatically).

Notably, one goal of the image interpretability scale is to rank-orderthe diagnostic tasks and criteria based on the interpretability of thedigital slide imagery. Accordingly, the selected fields of view(selected randomly or otherwise) preferably correspond to differentimage qualities, otherwise the scale will not have sufficient dynamicrange and it will not be possible to differentiate the relativedifficulty of achieving the criteria and tasks. In one embodiment,different image qualities for the same portion of a digital slide imagecan be obtained, for example, by sub-sampling the imagery data, bycompressing the imagery data (and further by changing the quality factorduring image compression), by scanning the glass slide at differentresolutions, any by other means for simulating different imagequalities.

The image scoring module 130 is configured to calculate, based on thesuccess or failure of task performance, an image interpretability score.The score is calculated for each portion of a digital slide image thatis examined by an analyst (or image processing algorithm). A separatescore may also be calculated for various regions of a digital slide anda further separate score may be calculated for the entire digital slide.In one embodiment, entire digital slide scores are calculated from therespective scores of individual portions of imagery data, or from therespective scores of individual regions of imagery data (comprising aplurality of portions), or from a combination of individual portions andregions of imagery data.

Advantageously, a single digital slide may have multiple scoresassociated with it, including scores for portions, regions, and theentire slide at the scanning resolution, as well as scores for portions,regions, and the entire slide at reduced intermediate resolutions allthe way down to the smallest resolution thumbnail image of the entireslide (and portions and regions thereof).

FIG. 4 is a flow diagram illustrating an example process for improveddiagnostic resolution for digital slide images according to oneembodiment of the present invention. In this embodiment, a first imageof a tissue sample is scanned at diagnostic resolution (230). The imageof the tissue sample can comprise a plurality of stains, for examplehematoxylin, eosin or DAB. The image of the tissue is deconvolved into aplurality of single images (240). Subsequently image adjustments aremade on one or more of the plurality of single stain images. The one ormore of the plurality of single stain images can then be combined into asecond image at diagnostic resolution (250), where the second image isan improved digital image. The improved digital image can be rated witha rating scale for image interpretability in anatomic pathology. In someembodiments the image adjustment (250) is performed, which may includedigital recolorization of a stain on a single stain image.Recolorization can provide higher contrast, personalization, identifyingevents/areas of interest (e.g., rare events, ER/PR co-location).

FIG. 5 is a flow diagram illustrating another example process forimproved diagnostic resolution for digital slide images according to oneembodiment of the present invention. In this embodiment a first image ofa tissue sample is scanned at diagnostic resolution (231). The tissuesample or image can comprise a plurality of stains, for examplehematoxylin, eosin or DAB. The image of the tissue is then deconvolvedinto a plurality of single images (241). The one or more of theplurality of single stain images are then processed with one or moreimage analysis algorithms or image quality algorithm (251). The imagequality algorithm can successfully separate the individual stains,digitally adjust each stain independently and recombine them in pairs topresent or store (261) the digital slides.

In one embodiment, N slides that contain serial sections of a tissueblock that are stained for different tests (for example H&E, ER, PR) canbe decomposed or stain separated, adjusted, recolored and the stainchannels can be combined from the different slides. Thus, instead ofseparating an H&E&DAB into an H&E and H&DAB the H&E and H&DAB can becombined to form H&E&DAB. The IQ algorithm can be configured with aviewing mode that allows a user, for example a pathologist, to overlayone view on another. For example when viewing during an H&E review, thepathologist can turn on the DAB view overlaying the H & E view. Anotherexample includes taking a combination of ER of H&DAB and PR of H&DABwhere both the DAB are brown and modifying the PR's DAB to green andcreating the ER of H&DAB with the PR's green DAB overlayed. In oneembodiment, the overlaying process can require image registrationbetween the N slides. The use of the IQ algorithm can result indigitally enhanced stains, a reduction in sample used, automaticco-location of multiple stains, for example H & E with H & DAB, therebyaccelerating review, saves preparation time and reduces the number ofglass slides and storage. The image quality algorithm can also configurefor any two or three stained sample, for example H & E with H & DAB or H& E & DAB. The image quality algorithm also performs digital stainadjustments for stain normalization including dilution/concentration,removal of background staining, cellular detail emphasis, chromogentuning and remapping. The data resulting from the image analysisalgorithms can be stored in memory 25, for example. The IQ Algorithm canbe configured to operate synchronously with viewing of the images orwith display updating. Thus the images resulting from the IQ algorithmcan be viewed or displayed (for example in viewing or display mode)without user intervention. Further the display updating feature canallow the images or stains to be updated synchronously with theoperation of the IQ algorithm. These operations of the IQ Algorithm arestain specific or test specific. The IQ Algorithm can be configured toprovide a digital viewing mode for viewing stains or images, including aside by side viewing.

FIG. 6 is an example of a flow process for enhanced progressiverendering when viewing digital slide images of a tissue according to anembodiment. This process may be implemented by a viewing station or animage server or a combination of these and other devices. Embodiments ofthe viewing station and/or image server can be implemented with ageneral purpose computer such as described below with respect to FIG. 18and modules that carry out the steps of the process may be implementedin software or hardware.

Initially, in step 100 a navigation instruction is received. Thenavigation instruction may identify a portion of a digital slide imageof a tissue that is desired to be viewed. Next, in step 125 a lowresolution image of the portion of the image to be viewed is obtained.This image data may be obtained from local memory or from an imageserver that is accessible via direct or indirect communication link. Forexample, an indirect communication link may comprise a networkconnection between a viewing station and an image server. In step 150the obtained image is displayed.

In step 175, a determination is then made as to whether the resolutionof the portion of the digital slide image of the tissue sample isacceptable based on an indicator. In one embodiment the indicator isindicated by a lapse of time. A predetermined threshold of time may beallocated for a displayed image with the acceptable resolution to beselected. Thus the time that the image is displayed is tracked and ifthe amount of time the image is displayed exceeds the predeterminedthreshold amount of time indicates that the resolution is unsatisfactoryor unacceptable. When such a determination is made a higher resolutionimage of the same area of interest is then obtained in step 225 if suchhigher resolution image exists, as determined in step 200.Alternatively, if the predetermined threshold amount of time is notexceeded before a subsequent navigation instruction is received, thenthe process loops back to step 100 where the next navigation instructionis received in order to view a different area of interest.

If there is a higher resolution image that is subsequently displayed instep 150, the process loops through progressive rendering of higherresolution images as long as it is desired to focus on that particulararea of interest. For example, a pathologist may focus on the particulararea of interest without issuing a new navigation instruction. Once thehighest resolution image has been displayed, if the pathologistcontinues to view the same area of interest, for example, then anenhanced image of the area of interest is obtained in step 250 anddisplayed in step 275. An enhanced image can be obtained according tothe illustration of FIG. 4 and FIG. 5 above. Obtaining the enhancedimage may include dynamically processing the current portion of theimage to enhance the image. Alternatively, obtaining the enhanced imagemay also include fetching a preprocessed enhanced image from datastorage.

Advantageously, an indicator may also be employed to signal to thepathologist that an enhanced image is being displayed, as shown in step300. Such an indicator may be visual such as an icon that is highlightedor brightened when an enhanced image is being displayed and is darkenedwhen a native image is being displayed. Alternatively, or incombination, an aural indicator may be used such as a sound thatindicates the current view is an enhanced image.

One advantage of dynamically processing the image data is that theentire digital slide does not need to be preprocessed with theenhancement but instead only the portion of the image under review maybe enhanced. As the pathologist navigates about the digital slide toregions of interest it is not always necessary to perform the imagequality improvement algorithm(s). Instead the on-demand dynamic approachcan be implemented such that when the pathologist dwells or focuses on aview, the enhanced view would be presented. Additionally, once anenhanced view has been created, for example in a dynamic embodiment, theenhanced view can be stored along with the digital image so thatsubsequent returns to the same area of interest will not requireredundant image processing in order to present the enhanced view.

Note that the improved progressive rendering described with respect toFIG. 6 is only one embodiment for improving image quality forpathologists. In progressive rendering, a sequence of low to mid to highresolution images are presented to the viewer until the highestresolution is displayed or until the user navigates away from that viewand restarts the rendering process. The improved progressive renderingmay add enhanced images at any point during the sequence of low to midto high resolution images, for example with the final presentation beingan enhanced high resolution image of the area of interest.

FIGS. 7-8 are block diagrams illustrating examples of images processedby an image quality algorithm. FIG. 7 shows an original image that isdecomposed into separate images, one for the H stain and one for the Estain. The H&E images are then recombined to provide an improved image.Similarly, FIG. 8 shows an original image broken down into threeseparate images that are then recombined into improved images havingvarious combinations. Although all combinations of the singledecomposed/adjusted images are not shown, all combinations can becreated.

FIGS. 9-12 are diagrams illustrating example regions examined andcollocation of markers and side by side viewing of the reconstructions.FIGS. 10-12 show a user interface having three views of the same region,including a view of the original digital image, a view of an enhanceddigital image having the brown stain turned off, and a view of anenhanced digital image having the pink stain turned off. In oneembodiment, turning off a stain can be accomplished by digitallyremoving one or more colors from the image.

FIG. 13 is a diagram illustrating an example flow of presenting IQAlgorithm images. Initially, the adjusted images are presented, then thedeconvolved images having individual stains are presented, then therecombined images (in any combination) are presented.

FIGS. 14-17 are diagrams illustrating examples of multiple image qualitypresentations of original images and recombined images having differentstains in various combinations (single or multiple stains).

FIG. 18 is a block diagram illustrating an example computer system 550that may be used in connection with various embodiments describedherein. For example, the computer system 550 may be used in conjunctionwith the server or reviewer devices previously described with respect toFIG. 1. However, other computer systems and/or architectures may beused, as will be clear to those skilled in the art.

The computer system 550 preferably includes one or more processors, suchas processor 552. Additional processors may be provided, such as anauxiliary processor to manage input/output, an auxiliary processor toperform floating point mathematical operations, a special-purposemicroprocessor having an architecture suitable for fast execution ofsignal processing algorithms (e.g., digital signal processor), a slaveprocessor subordinate to the main processing system (e.g., back-endprocessor), an additional microprocessor or controller for dual ormultiple processor systems, or a coprocessor. Such auxiliary processorsmay be discrete processors or may be integrated with the processor 552.

The processor 552 is preferably connected to a communication bus 554.The communication bus 554 may include a data channel for facilitatinginformation transfer between storage and other peripheral components ofthe computer system 550. The communication bus 554 further may provide aset of signals used for communication with the processor 552, includinga data bus, address bus, and control bus (not shown). The communicationbus 554 may comprise any standard or non-standard bus architecture suchas, for example, bus architectures compliant with industry standardarchitecture (“ISA”), extended industry standard architecture (“EISA”),Micro Channel Architecture (“MCA”), peripheral component interconnect(“PCI”) local bus, or standards promulgated by the Institute ofElectrical and Electronics Engineers (“IEEE”) including IEEE 488general-purpose interface bus (“GPIB”), IEEE 696/S-100, and the like.

Computer system 550 preferably includes a main memory 556 and may alsoinclude a secondary memory 558. The main memory 556 provides storage ofinstructions and data for programs executing on the processor 552. Themain memory 556 is typically semiconductor-based memory such as dynamicrandom access memory (“DRAM”) and/or static random access memory(“SRAM”). Other semiconductor-based memory types include, for example,synchronous dynamic random access memory (“SDRAM”), Rambus dynamicrandom access memory (“RDRAM”), ferroelectric random access memory(“FRAM”), and the like, including read only memory (“ROM”).

The secondary memory 558 may optionally include a hard disk drive 560and/or a removable storage drive 562, for example a floppy disk drive, amagnetic tape drive, a compact disc (“CD”) drive, a digital versatiledisc (“DVD”) drive, etc. The removable storage drive 562 reads fromand/or writes to a removable storage medium 564 in a well-known manner.Removable storage medium 564 may be, for example, a floppy disk,magnetic tape, CD, DVD, etc.

The removable storage medium 564 is preferably a computer readablemedium having stored thereon computer executable code (i.e., software)and/or data. The computer software or data stored on the removablestorage medium 564 is read into the computer system 550 as electricalcommunication signals 578.

In alternative embodiments, secondary memory 558 may include othersimilar means for allowing computer programs or other data orinstructions to be loaded into the computer system 550. Such means mayinclude, for example, an external storage medium 572 and an interface570. Examples of external storage medium 572 may include an externalhard disk drive or an external optical drive, or and externalmagneto-optical drive.

Other examples of secondary memory 558 may include semiconductor-basedmemory such as programmable read-only memory (“PROM”), erasableprogrammable read-only memory (“EPROM”), electrically erasable read-onlymemory (“EEPROM”), or flash memory (block oriented memory similar toEEPROM). Also included are any other removable storage units 572 andinterfaces 570, which allow software and data to be transferred from theremovable storage unit 572 to the computer system 550.

Computer system 550 may also include a communication interface 574. Thecommunication interface 574 allows software and data to be transferredbetween computer system 550 and external devices (e.g. printers),networks, or information sources. For example, computer software orexecutable code may be transferred to computer system 550 from a networkserver via communication interface 574. Examples of communicationinterface 574 include a modem, a network interface card (“NIC”), acommunications port, a PCMCIA slot and card, an infrared interface, andan IEEE 1394 fire-wire, just to name a few.

Communication interface 574 preferably implements industry promulgatedprotocol standards, such as Ethernet IEEE 802 standards, Fiber Channel,digital subscriber line (“DSL”), asynchronous digital subscriber line(“ADSL”), frame relay, asynchronous transfer mode (“ATM”), integrateddigital services network (“ISDN”), personal communications services(“PCS”), transmission control protocol/Internet protocol (“TCP/IP”),serial line Internet protocol/point to point protocol (“SLIP/PPP”), andso on, but may also implement customized or non-standard interfaceprotocols as well.

Software and data transferred via communication interface 574 aregenerally in the form of electrical communication signals 578. Thesesignals 578 are preferably provided to communication interface 574 via acommunication channel 576. Communication channel 576 carries signals 578and can be implemented using a variety of wired or wirelesscommunication means including wire or cable, fiber optics, conventionalphone line, cellular phone link, wireless data communication link, radiofrequency (RF) link, or infrared link, just to name a few.

Computer executable code (i.e., computer programs or software) is storedin the main memory 556 and/or the secondary memory 558. Computerprograms can also be received via communication interface 574 and storedin the main memory 556 and/or the secondary memory 558. Such computerprograms, when executed, enable the computer system 550 to perform thevarious functions of the present invention as previously described.

In this description, the term “computer readable medium” is used torefer to any media used to provide computer executable code (e.g.,software and computer programs) to the computer system 550. Examples ofthese media include main memory 556, secondary memory 558 (includinghard disk drive 560, removable storage medium 564, and external storagemedium 572), and any peripheral device communicatively coupled withcommunication interface 574 (including a network information server orother network device). These computer readable mediums are means forproviding executable code, programming instructions, and software to thecomputer system 550.

In an embodiment that is implemented using software, the software may bestored on a computer readable medium and loaded into computer system 550by way of removable storage drive 562, interface 570, or communicationinterface 574. In such an embodiment, the software is loaded into thecomputer system 550 in the form of electrical communication signals 578.The software, when executed by the processor 552, preferably causes theprocessor 552 to perform the inventive features and functions previouslydescribed herein.

Various embodiments may also be implemented primarily in hardware using,for example, components such as application specific integrated circuits(“ASICs”), or field programmable gate arrays (“FPGAs”). Implementationof a hardware state machine capable of performing the functionsdescribed herein will also be apparent to those skilled in the relevantart. Various embodiments may also be implemented using a combination ofboth hardware and software.

Furthermore, those of skill in the art will appreciate that the variousillustrative logical blocks, modules, circuits, and method stepsdescribed in connection with the above described figures and theembodiments disclosed herein can often be implemented as electronichardware, computer software, or combinations of both. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps have beendescribed above generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled persons can implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the invention. In addition, the grouping of functions within amodule, block, circuit or step is for ease of description. Specificfunctions or steps can be moved from one module, block or circuit toanother without departing from the invention.

Moreover, the various illustrative logical blocks, modules, and methodsdescribed in connection with the embodiments disclosed herein can beimplemented or performed with a general purpose processor, a digitalsignal processor (“DSP”), an ASIC, FPGA or other programmable logicdevice, discrete gate or transistor logic, discrete hardware components,or any combination thereof designed to perform the functions describedherein. A general-purpose processor can be a microprocessor, but in thealternative, the processor can be any processor, controller,microcontroller, or state machine. A processor can also be implementedas a combination of computing devices, for example, a combination of aDSP and a microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration.

Additionally, the steps of a method or algorithm described in connectionwith the embodiments disclosed herein can be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of storage mediumincluding a network storage medium. An exemplary storage medium can becoupled to the processor such the processor can read information from,and write information to, the storage medium. In the alternative, thestorage medium can be integral to the processor. The processor and thestorage medium can also reside in an ASIC.

The above description of the disclosed embodiments is provided to enableany person skilled in the art to make or use the invention. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles described herein can beapplied to other embodiments without departing from the spirit or scopeof the invention. Thus, it is to be understood that the description anddrawings presented herein represent a presently preferred embodiment ofthe invention and are therefore representative of the subject matterwhich is broadly contemplated by the present invention. It is furtherunderstood that the scope of the present invention fully encompassesother embodiments that may become obvious to those skilled in the artand that the scope of the present invention is accordingly limited bynothing other than the appended claims.

The invention claimed is:
 1. A computer-implemented method for improvingthe image quality of a digital slide image in a digital environment forthe management and interpretation of pathology information, the methodcomprising: receiving a digital slide image of a tissue sample atdiagnostic resolution, wherein the tissue sample has been stained usingthree or more stain types; deconvolving the digital slide image intothree or more single images having one of the three or more stain typesper single image; processing the three or more single images with animage quality algorithm to optimize the quality of the three or moresingle images; and recombining two or more of the optimized three ormore single images into two or more recombined digital slide images atdiagnostic resolution, wherein each of the two or more recombineddigital slide images comprises a different combination of two or moresingle images than others of the two or more recombined digital slideimages, and wherein at least one of the two or more recombined digitalslide images comprises a combination of at least two, but not all, ofthe three or more single images.
 2. The method of claim 1, furthercomprising storing the three or more optimized single images in astorage device.
 3. The method of claim 1, further comprising adjustingone or more of the three or more single images.
 4. The method of claim3, wherein the adjusting includes digital recolorization of a stain inthe one or more single images.
 5. The method of claim 1, wherein thereceiving comprises scanning the digital slide image at diagnosticresolution.
 6. The method of claim 1, wherein the image properties ofone or more of the three or more single images are linked with theproperties of a light passing through the tissue sample and onto thepixel sensors of a device for recording the images.
 7. The method ofclaim 1, wherein the image quality algorithm is configured to performdigital stain adjustments for stain normalization.
 8. The method ofclaim 7, wherein the digital stain adjustment process is selected fromthe group of processes consisting of dilution/concentration, removal ofbackground staining, cellular detail emphasis, chromogen tuning andremapping.
 9. A computer-implemented method for improving the imagequality of a digital slide image in a digital environment for themanagement and interpretation of pathology information, the methodcomprising: receiving a digital slide image of a tissue sample atdiagnostic resolution, wherein the tissue sample has been stained usingthree or more stain types; deconvolving the digital slide image intothree or more single images having one of the three or more stain typesper single image; adjusting one or more of the three or more singleimages with an image analysis algorithm to optimize the quality of theone or more single images; and recombining two or more of the three ormore single images, including the adjusted one or more single images,into two or more recombined digital slide images at diagnosticresolution, wherein each of the two or more recombined digital slideimages comprises a different combination of two or more single imagesthan others of the two or more recombined digital slide images, andwherein at least one of the two or more recombined digital slide imagescomprises a combination of at least two, but not all, of the three ormore single images.
 10. The method of claim 9, wherein the receivingincludes scanning the digital slide image at diagnostic resolution. 11.A system for improving the image quality of a digital slide image in adigital environment for the management and interpretation of pathologyinformation comprising: a receiving module configured to receive adigital slide image of a tissue sample at diagnostic resolution, whereinthe tissue sample has been stained using three or more stain types; adeconvolution module configured to deconvolve the digital slide imageinto three or more single images having one of the three or more staintypes per single image; a controller module configured to execute ananalysis algorithm for processing one or more of the three or moresingle images to optimize the quality of the one or more single images;and a recombination module configured to recombine two or more of thethree or more single images, including the adjusted one or more singleimages, into two or more recombined digital slide images at diagnosticresolution, wherein each of the two or more recombined digital slideimages comprises a different combination of two or more single imagesthan others of the two or more recombined digital slide images, andwherein at least one of the two or more recombined digital slide imagescomprises a combination of at least two, but not all, of the three ormore single images.
 12. The system of claim 11, further comprising astorage device for storing the three or more single images.
 13. Thesystem of claim 11, wherein the controller module is further configuredto make an adjustment to one or more of the three or more single images.14. The controller module of claim 13, wherein the adjustment includesdigital recolorization of a stain in the one or more single images. 15.The system of claim 13, wherein the adjustment is predefined for a firsttest and automatically applied.
 16. The system of claim 13 wherein thecontroller module is further configured to receive adjustmentinstructions from a user for the current instance of viewing.
 17. Thesystem of claim 11, wherein the receiving module comprises a scannermodule configured to scan a digital slide image of a tissue sample atdiagnostic resolution.